The invention relates to a method for the detection of fog by use of a camera image or a video image.
The occurrence of fog represents a significant impairment of the movement of goods and persons on the ground, in the air and at sea. Modern vehicles can therefore be equipped with automated video-based fog detection systems which may be coupled with different driver assistance systems.
For example, in N. Hautiere et al. “Automatic Fog Detection and Estimation of Visibility Distance through Use of an Onboard Camera”, Machine Vision and Applications, Vol. 17, No. 1, pp. 8-20, January 2006, an approach is described for the video-based fog detection and visibility distance estimation in a vehicle during daytime, which is also the object of French Patent Document FR 2835911 B. In this case, an image is processed which was taken by a front camera. By means of a region-growing approach, an image area is extracted which contains parts of the road as well as parts of the sky. Within this area, a measuring range is defined which contains the median of the intensity values for every image line. The (intensity) course from the lowermost to the uppermost image line of this measuring range is called an intensity function. The turning point of the intensity function defines a visibility distance, i.e. the image line above which objects are no longer perceptible to the driver. The turning point of the intensity function is defined by the image line above which the driver can no longer recognize any objects. By way of the perspective projection and assuming that a topography is flat, a corresponding visibility distance in meters can be inferred for this image line. When the image line having the turning point is situated above the image center point, a visibility distance of infinite is obtained, whereby, by means of this process, the presence of fog can also be recognized.
However, in “Perception through Scattering Media for Autonomous Vehicles”, Autonomous Robots Research Advances, Nova Science Publishers, pp. 223-267, April 2008, Hautiere et al. themselves determined that the originally used region-growing approach for the extraction of the image region to be investigated, which contains parts of the road as well as parts of the sky, supplies insufficient results in many traffic situations. It cannot always be guaranteed that the horizon is present in the form of a transition from the road to the sky, as, for example, in the case of traffic driving ahead or in the case of certain landscape-related circumstances.
An alternative approach is the topic of Bronte et al. in “Fog Detection System Based on Computer Vision Techniques”, IEEE Intelligent Transportation Systems, pp. 1-6, October 2009, with a system for the detection of fog and for the estimation of the visibility distances. In this case, two adjoining regions—a road area and a sky area—are extracted in a camera image by way of a region-growing algorithm. Furthermore, the actual vanishing point is calculated by way of image characteristics. In the case of a fog scene, the sky area, measured from the uppermost image line, is higher than the vanishing point. By way of the image line, in which the two regions touch one another, the visibility distance can be determined by way of the perspective projection and the assumption of a flat topography. In order to avoid faulty detections, the algorithm is additionally preceded by a threshold-value-based “no-fog detector”. The latter detects, by means of the sum of the occurring gradient amounts within the upper image half whether the image is sufficiently blurred in order to carry out an estimation of the visibility distance. Since the estimation of the visibility distance based on a single image is very unreliable, i.e. with respect to time, is subjected to strong fluctuations over several successive images, a median formation is implemented over a fixed time period.
It is an object of the invention to describe an improved method for the detection of fog by use of a camera image or a video image.
This and other objects are achieved by a method for the detection of fog by use of a camera image or a video, the method comprising the steps of: (a) taking a two-dimensional image with at least one color channel or several color channels as a function of two independent location coordinates, (b) determining a two-dimensional gray-scale function for the at least one color channel or for each of the several color channels, which defines the value of the gray scale as a function of the two independent location coordinates of the two-dimensional image, and (c) two-dimensional Fourier transforming of the two-dimensional gray-scale function depending on two independent frequency coordinates.
The method according to the invention comprises the following steps.
First, a two-dimensional image is taken with at least one color channel or several color channels, and a gray-scale image is created for each of the several color channels. The value of the gray stage image is a function of two independent location coordinates as a two-dimensional gray-scale function.
Subsequently, the gray-scale function is two-dimensionally Fourier-transformed, the Fourier transform being a function of two independent frequency coordinates in the frequency domain.
In the case of this method, the circumstance is utilized that, when taking a gray-scale image of a scene in fog, the intensity differences or gray-scale value gradients of neighboring points will be smaller as the fog becomes increasingly dense than when taking a gray-scale image of the same scenery without any fog. In the image without any fog, sharper edges and higher contrasts can be clearly recognized than in the case of the setting with fog. For example, a white road marking on an asphalt surface becomes indistinct in fog, whereas it can be recognized in a sharply contoured shape in sunshine.
A Fourier transformation of, for example, a one-dimensional step function in the frequency domain supplies different contributions also at high frequencies of zero, which disappear, for example, in the case of the Fourier analysis of a one-dimensional Gaussian distribution curve.
The Fourier transformation therefore has the advantage that the “degree” of the occurrence of fog is scaled in the frequency domain: In the case of fog, the amount of the Fourier transform is concentrated about the zero point of the frequency coordinates; in sunshine, the Fourier transformation increasingly supplies contributions at values of the two frequency coordinates whose amounts are high. The method according to the invention is therefore particularly advantageous for detecting fog by use of camera or video still images.
An image in a black-white representation is based on a single color channel. In contrast, an image in color is normally based on three color channels—a red channel, a green channel and a blue channel. An intensity distribution is assigned to each channel of the image, which intensity distribution can be shown as a gray-scale image.
According to a preferred variant of the method, the two-dimensional gray-scale image is scaled.
A scaling of the gray-scale image has the advantage that local image regions, if possible, are equalized, for example, by illumination and lighting effects. The scaling can be achieved by suitable filters, such as isotropic high-pass and low-pass filters. The scaling prevents the dominance of some image regions in the power spectrum. As an example, cast shadows on a surface that is homogeneous per se can be mentioned whose influence can be reduced by the scaling.
Furthermore, the square of the absolute value of the Fourier transform is calculated, which is called the power spectrum. In the two-dimensional frequency domain, a digital image processing is carried out at the power spectrum.
According to a preferred embodiment of the invention, the power spectrum is analyzed by using a family of Gabor filters. In this case the power spectrum is filtered by means of each individual Gabor filter of the family of Gabor filters over the two-dimensional frequency domain and the result of the filtering is called a Gabor characteristic.
The power spectrum is “scanned” by use of a point-symmetrical family of Gabor filters in the two-dimensional frequency domain. This means that the image set of the power spectrum is also used as the image set for the family of Gabor filters. Each individual Gabor filter, whose image area and whose value area in the frequency domain is defined by its scaling and orientation, is folded with the power spectrum. The folding value is called a Gabor characteristic. In this manner, a Gabor characteristic is assigned to each Gabor filter, and the contribution of the power spectrum in various frequency ranges of the frequency domain is made quantitatively measurable.
In addition, the Gabor characteristics are multiplied by a predefined weighting function in order to calculate a fog indicator. The fog indicator is compared with a specified threshold value. As a result of the comparison, a specified value for “fog” or another specified value for “no-fog” is assigned to a binary variable with a confidence measurement. This means that the process provides the information “fog” or “no-fog” with a confidence measurement.
In addition, it is advantageous for the specified weighting function and the specified threshold value to be determined by way of empirical learning data.
This indicates that, during the development, i.e. before being applied to the intended use, the process is empirically validated by use of learning and training data. For this purpose, by use of test images with visibility distance estimation, an objectivized fog classification of the taken images can be carried out in order to evaluate the process by use of these data.
According to an alternative variant of the present invention, by use of Gabor characteristics, a reduction of characteristics can be carried out with a main component analysis. The reduced characteristics are called main Gabor characteristics.
A main component analysis has the advantage that the Gabor characteristics can be statistically examined for their relevance by means of training data. This permits a resource-saving management of computing power when the process is used as intended.
According to a further variant of the present invention, the power spectrum can be subjected to a reduction of characteristics with a main component analysis. The reduced characteristics are called main characteristics.
In the case of the digital image processing process of the component analysis, each pixel value of the power spectrum in the frequency domain is considered to be an individual characteristic. On the basis of these characteristics, a reduction of characteristics can be carried out by the main component analysis. As an alternative, a support vector process can be used in the frequency domain.
According to a further embodiment of the invention, the Gabor characteristics or the main Gabor characteristics are classified. The classification is carried out by a linear classifier or a non-linear classifier. During the classification, a value and a confidence measurement are assigned to the classification variable which correlates with the occurrence and, optionally, additionally the density of fog.
Here, the linear discriminant analysis or the support vector process, for example, known to the person skilled in the art for digital image processing, can be used. Finally, the classification process outputs a class affiliation, such as “fog” or “no fog” as information. This class affiliation may also be provided with a confidence measurement.
It is particularly advantageous that, in a vehicle, which includes a control unit, a camera system or video system and at least one driver assistance system, the method for the detection of fog is carried out by the camera or video system and by the control unit in real-time, and the class affiliation is transmitted to the at least one driver assistance system. The driver assistance system of the vehicle can output a warning to the driver in case of “fog”. As an alternative or in addition, the driver assistance system can be operated in a configuration specified for “fog”. A direct contribution can thereby be made to active safety in road traffic.
In this context, real-time means that the information of the class affiliation can be determined by way of the process within a time window which is shorter with respect to time than the inverse image-taking frequency of the gray-scale images of the camera or video system.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings.